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Abstract 



The concept of EPSS (Electronic Performance Support System) originated in business settings. Recently, 
there have been many attempts to apply the concept to schools: educational EPSSs (E-EPSSs) have become 
available on the Web. However, there is little evaluation research and few evaluation frameworks for these 
emergent E-EPSSs. The primary purpose of this article is to provide our design recommendations for how 
to improve the quality of E-EPSSs in general, based upon the evaluation of one specific E-EPSS, called 
STEPS (Support for Teacher Enhancing Performance in Schools). To achieve this purpose, the article first 



reviews E-EPSSs in terms of teachers ' professional development and discusses their encompassing trends, 
needs, and definitions. Secondly, it presents an evaluation case of STEPS. An evaluation perspective called 
** perception -oriented usability evaluation’’ drives the evaluation. Lastly, it lists recommendations for 
improving STEPS as well as E-EPSSs in general based on our STEPS evaluation results and literature 



review. 



Introduction 

An exponential increase in information requires teachers to continuously develop their professional skills. 
As a response to this requirement, many researchers have proposed creating an EPSS (Electronic Performance 
Support System) to support instructional design activities, which is one of the main tasks of the teachers, through 
job-embedded learning (Reigeluth, 1999; Gustafson, 1993), and to promote training in education (Scales, 1994). 
EPSS has also been acknowledged as a system that can assist the school reform movement rather than a mere tool 
that may exert its influence in only a piecemeal way (Northrup & Pilcher, 1998; Scale, 1994). Applying an EPSS 
systemically not only can alleviate instructional and administrative burdens by supporting teacher performance, but 
also can provide teachers with job-embedded training opportunities. Using an EPSS, teachers can receive training 
within their teaching context. They don’t necessarily need to leave their classrooms and school environment to 
improve their professional skills. 

The concept of EPSS originated in business settings; however, recently, there have been many attempts to 
apply the concept to schools (Barker & Benerji, 1995; Northrup & Pilcher, 1998), and some educational EPSSs (E- 
EPSSs) have become available on the Web. However, there is little evaluation research and correspondingly few 
evaluation frameworks for these emergent E-EPSSs. In response to this scarcity of evaluation research, this article 
describes an evaluation study on one E-EPSS, called STEPS (Support for Teacher Enhancing Performance in 
Schools.) The article is composed of the following sections: 

• Overview of the general features of a selected E-EPSS, STEPS. 

• Review of E-EPSS as embedded in teacher professional development and its encompassing trends, 
needs, and definitions. 

• Description of evaluation methodology. 

• Results of the STEPS evaluation. 

• Recommendations for the improvement of STEPS and the design of E-EPSS in general. 

STEPS as an Evaluand 

STEPS (Support for Teacher Enhancing Performance in Schools) is an EPSS designed specifically to help 
pre-service and practicing pre-K-12 teachers develop instructional lessons, units, and curricula aligned to Florida's 
Sunshine State Standards. According to Northrup and Pilcher (1998), the purpose of STEPS is to support school 
reform and sustain accountability of the integrated curriculum that utilizes technology, alternative assessment, and 
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diverse learning environments and its conceptual frameworks include flexibility, learning by doing, and a user- 
designed structure. 

Figure /. The screen shots of STEPS 
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STEPS home page 



Main page for the grades 3-5 teachers 



STEPS is available through the World Wide Web or as a standalone CD-ROM. The web version of the 
EPSS was selected for this study. The first picture above is the screen shot of the home page of STEPS. It consists 
of four menus broken down by the grade level: PK-2, 3-5, 6-8, and 9-12. If the user clicks the signpost marked 3-5, 
then the main page for the grades 3-5 teachers, which is depicted in the second picture, opens. The main page of 
grades 3-5 consists of two groups of menus: the left -hand-side frame contains links to the Main Menu, which 
includes Lesson Architect, Tutorial Library, Best Practices, Sample Unit, Web Links, State Standards, Florida 
Information Resources Network (FIRN), and Florida Department of Education (FDE), and the right-hand-side frame 
includes links to the same menus in the left-hand frame. The main page for each grade group has the same menu 
structure. 

The main six components of STEPS are Lesson Architect, Tutorial Library, Best Practices, Sample Unit, 
Web Links, and Coach. Lesson Architect is the main component of STEPS. It guides teachers through the processes 
of instructional design and curriculum planning. It uses Gagne’s Events of Instruction (1992) and Dick and Carey’s 
Systematic Instructional Design Models (1996) as theoretical foundations of instructional design, and various 
curriculum approaches such as webbing and threaded curriculum (Northrup & Pilcher, 1998). 

Tutorial Library is a collection of about forty instructional tutorials. The tutorials follow the four premises 
of the STEPS curriculum: integrated curriculum, integrated technology, alternative assessment, and diverse learning 
environment (Northrup & Pilcher, 1998). Some tutorials are linked from the Lesson Architect so that users can get 
tutorials while they are planning lessons using the Lesson Architect. This function demonstrates one of the main 
advantages of EPSS -just-in-time support to the users. 

Best Practices provide a forum for sharing successful classroom activities that were developed and tested 
by teachers in their real classrooms. In addition to keyword searches, users can search activities by Sunshine State 
Standard item or by theme. They can also browse by subject areas. This provides instructional strategies and tactics 
for designing classroom activities. 

Sample Units provide sample curriculum units created by teachers of the same grade level. Grades 3-5 
sample units, for example, center around an archaeological theme and provide 10-day model curriculum units. The 
model units utilize teachers’ cross-curricular connections in Math, Science, Social Studies, and Language Arts and 
follow the benchmarks described in Florida’s Sunshine State Standards. The teachers used Lesson Architect to 
create the units. 

Web Links include over 400 web sites relevant to Math, Science, Social Studies, and Language Art 
identified in Florida's Sunshine State Standards. Each web site is linked to a brief description of the website and a 
list of applicable benchmarks. 

Coach offers three levels of scaffolding: the "big picture" level, the "what do I do" level, and the "how do I 
do" level. It models the Knowledge Integration Environment created by the University of California, Berkeley 
(Bell, Davis, & Lynn, cited in Northrup & Pilcher, 1998). If users click “Ask Coach” or a corresponding graphic 
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icon, a pop-up window appears and provides screen- and field -sensitive help such as descriptions of what the Lesson 
Architect is or how users can search information on relevant screens. Gery (1991) calls it an extrinsic support that 
is integrated with the software but not a primary function. 

Literature Review 

E-EPSS as a tool for teacher professional development 

Traditional professional development programs for teachers have stressed knowledge acquisition through 
workshops and courses. These programs have had difficulties in providing sustained support needed for teachers to 
apply what they have learned in their classrooms . The performance-centered approach, however, provides such a 
sustained support by employing more practical knowledge dissemination. Teachers engage in performing processes 
through which they find problems, organize information, and infer a series of proper decision-making activities 
needed to solve their classroom problems. Specifically, the new professional development approach engages 
teachers in a series of concrete tasks of teaching, assessment, observation, and reflection. It is grounded in 
participant-driven inquiry, reflection, and experimentation. It supports collaboration and knowledge sharing among 
teachers, and focuses on communities of teaching practice rather than on individual teachers. It is intensive and 
sustained and supported by modeling, coaching, and collective problem-solving. Finally, it attempts to relate itself 
to other aspects of organizational change. 

As a tool for the performance-centered approach and one of the viable alternatives for traditional 
professional development, an EPSS has been proposed for supporting the most critical activities of teachers, 
instructional design activities, i.e., curriculum development and lesson design. EPSS has been defined in various 
ways; however, there exist general agreements on the major goals of an EPSS. The goals are to: (1) provide 
"whatever is necessary to generate performance and learning at the moment of need," referred to as "just-in-time 
training systems" (Geber, 1991, p.34); (2) enable "day-one performance," the idea that novice users should be 
productive on the very first day that they start using a system (Gery, 1995); and (3) support higher levels of 
performance for the work being done today, while helping to build the knowledge infrastructure for work to be done 
in the future (Winslow & Bramer, 1994). 

In the education field, there have been many electronic performance applications that meet or closely meet 
these common goals of EPSS although they are not always given the label “EPSS” (Collis & Verwijs, 1 995). The 
applications include instructional material development tools, grade books, and behavioral management support 
systems. What are the differences between these electronic performance applications for teachers and E-EPSS? E- 
EPSS is an “integrated” support system that includes tools, expert systems, instructional activities, and databases 
that assist teachers “at the time and place they need the assistance.” By supporting teacher practices at the moment 
of need, it helps teachers to improve their professional skills. The electronic performance applications, on the other 
hand, are single tools that support teachers in performing specific tasks. They are developed primarily as 
supplementary instructors, rather than as teacher professional development tools, that usually assist students’ 
learning activities. In addition, they are based on traditional classroom practices that assume education is a delivery 
or transfer of knowledge (Chiero, 1996; Collis & Verwijs, 1995). 

The recent advance of the Internet and other computer technology has created opportunities for teachers to 
use various Internet-based communication functions such as emails, distribution lists, and forums. Using these 
tools, teachers can now consult with experts in the areas where they have questions and share their knowledge and 
experiences with other teachers effectively and efficiently. Several examples of EPSSs that utilize such 
communication functions are currently available on the web. The STEPS web site available at 
http.7/143.88.86.98/pacee/steps/welcome.cfm (or at http://www.ibinder.uwf.edu/steps/welcom.cfm) is an EPSS 
designed specifically to help K-12 teachers to develop instructional lessons, units, and curricula. Another example. 
Pathways to School Improvement, available at http://www.ncrel.org/sdrs/pathwavg.htm . is designed primarily to 
assist high school teachers with their curriculum development. 

E-EPSS Design 

Many design guidelines and strategies for general EPSS can be applied to the design of E-EPSS. However, 
in designing E-EPSS, special attention must be paid to reflecting teachers’ unique professional characteristics onto 
the system. This section discusses design suggestions adopted from the general EPSS design literature. 

Components of E-EPSS. 
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Leighton (1997) synthesizes the ideas of Geiy (1991, 1995) and Raybould (1990), and contends that an 
EPSS has four typical components: tools, an information base, an advisor, and learning experiences. Similarly, Carr 
(1992) explains that an EPSS can play the roles of a librarian, an advisor, and an instructor. Even though researchers 
name components of E-EPSS differently, their classifications have commonalties. Typically, an E-EPSS is 
composed of tools, a database, an expert system, and instructions (Gery, 1991; Raybould, 1990). The following list 
summarizes functions and examples of each component: 

• Tools: Tools usually embody recommended procedures or best practices that should be employed 
by the user (Reeves, 1995). Examples of tools include word processing, spreadsheets , templates, 
and forms . 

• Database: A dynamic EPSS includes an infobase and users supply much of its content. The shared 
base of user experiences grows over time, making the infobase increasingly valuable to 
organizations (Laffey, 1995). An infobase may contain on-line documents, reference material, 
case history data, etc. 

• Expert system: An expert system usually provides two distinct types of support: proactive support 
and reactive support. Proactive support is usually delivered through a coach that provides 
assistance in setting goals and monitoring task completion. Reactive support is delivered through 
context-sensitive on-line help that assists users when they have reached an impasse and cannot 
proceed without overcoming a problem in using the software. 

• Instructions: Instructions typically include Computer-Based Training (CBT), but not in a 
traditional form of CBT. Traditional CBT might employ a sequential approach in providing a 
four-hour course on instructional design process. Learning experiences within an EPSS, however, 
must be organized into capsules that contain five to fifteen minutes of instruction. The capsules 
typically deal with specific topics that can be accessed while tasks are being performed. Examples 
of instructions include multimedia, CBT, tutorials, simulations, and scenarios. 

Depending on the scope and nature of an EPSS and technological platform, an EPSS is made up of a 
combination of at least one or more of these four components. 

Interface Design of E>EPSS. 

User interface is the single most important element of a successful electronic performance support system 
(Gery, 1995; Cole, Fisher & Saltzman, 1997). It is important to design the interface of an E-EPSS in a way that will 
support teachers’ performance (Law, Okey, & Carter, 1995). To do so, designers need to consider teachers’ mental 
models about teaching (i.e., what teachers think about instruction), workflow, and daily activities. The interface that 
follows teachers* natural workflow using screen metaphors that are familiar to them, facilitates understanding of the 
functions of an EPSS and accordingly reduces time needed for training. Hansen and Perry (1993) argue that long- 
term success of a system depends on teachers’ degree of comfort and confidence in using a system. 

As ways to capture teachers’ mental models. Law et al. (1995) suggest the case-based reasoning approach 
through which one can analyze complex problems. They also recommend conducting a task analysis to determine 
what task components a product should include and how each component contributes to the overall product. To 
identify necessary components to be included, it is necessary to analyze the actual daily-based tasks, the 
performance of these tasks, and the elements that can alter each task (Moore, 1998). The analysis techniques of 
EPSS task analysis are similar to those of ISD (Witt & Wager, 1 994). They include interviews, observations, 
questionnaires, and small -group discussions. EPSS task analysis, however, requires collecting data from both 
experts and novices, especially the data on their cognitive work processes, unique professional demands, and job- 
specific situations (Villachica& Stone, 1999; Stevens & Stevens, 1995). In other words, the components and 
content of an EPSS for teachers need to reflect teachers’ unique workload in different types of schools - 
kindergarten, elementary school, and secondary school. Orey, Moore, Hardy, and Serrano (1997) report that middle 
school teachers spend an average of 31.6 minutes per day preparing resources and 49.4 minutes per day planning 
lessons. Depending on types of school categories of teachers’ workloads and proportion of time, time breakdown is 
somewhat different. However, they are similar in that deskwork (i.e., grading) consumes a large portion of the 
teachers’ office time. 

The structure of an EPSS should be easy to use, flexible, and tailored for end-users having different needs 
and different expertise (Hansen & Perry, 1993; Remmers, 1998). A solution to accommodate different levels of 
competency of teachers is not to make all information directly visible, but to make it accessible, for instance by 
search tools that enable end users to find precisely the information they need (Sherry & Wilson, cited in Remmers, 
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1998). The structure of an EPSS must be customizable for the needs of different districts, schools, and teachers. In 
addition, cognitive loads and relational maps can be considered in design. 

Raybould (cited in Gery, 1991) suggests six effective screen structures for different information maps: 
single frame, tree, network, linear format, rule, and animation. The Yale Style Manual (1999) introduces structures 
of Web-based EPSS that include sequence, grid, hierarchy, web structure, and empirical structure. Since each 
structure has its own strengths and needs, the developer can choose the system which best matches the prospective 
users’ characteristics. For example, empirical structure is an effective way to organize a less-abstract view of the 
content when the users are novices in the field (Remmers, 1998). 

Carroll’s (1998) study on text interface led to the design of minimal manuals that drastically cut verbiage, 
encourage active involvement with hands on experiences as soon as possible, and promote guided exploration of 
system features. According to Paivio (1990, 1991), clear texts or images only are better sometimes than unclear texts 
with images and vice versa. To employ graphics in the design of EPSS, it is necessary to understand how graphics 
and other media (e.g., texts, audio, and videos) are cognitively processed and affect learning. The dual coding 
theory by Paivio (1990, 1991) describes how graphics become associated with texts in space and time. It also 
describes ways to organize materials according to the students’ previous experiences. 

Lastly, an action-oriented interface should anchor tools in the corresponding task domains and support error 
recognition and recovery in addition to the users’ performance. 

Usability Evaluation 

Amidst the rise of the user-centered design principle, many system design institutions have realized the 
importance of usability evaluation and are practicing it as an ongoing system design and development process. 

The formal definition of usability written by International Standards Organization (ISO, 9241 -1 1 ) is ’’the 
effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in a particular 
environment" (Bevan & Macleod, 1 994, p. 1 35). Bevan and Macleod explain that this definition characterizes 
usability as a "quality of use that can be measured as the outcome of interaction in a context" (p.l35). What they 
mean by this is that the overall system context is composed of users, their goals (or tasks), and the physical and 
organizational environment, as well as the system itself The quality of use is indicated by the degree of 
effectiveness, efficiency, and satisfaction as experienced in the result of the interaction among the four system 
components. The first two usability indicators, effectiveness and efficiency, are usually assessed by collecting 
behavioral performance measures in regard to leamability, efficiency, productivity, memorability, and number of 
errors (Nielsen, 1993). Satisfaction, on the other hand, is usually assessed by examining users’ perceptions about 
the system. 

Perceptions affect every aspect of the system, including usability. It is not rare to hear from users that they 
think a certain system is usable when they actually failed to perform tasks using the system. The reverse is often 
heard, too. This discrepancy between actual performance and perceptions has not been often researched in the field 
of usability evaluation. Traditionally, usability studies tended to collect only performance measures and determine a 
system's overall usability based on them. Even worse, they tended to disregard users’ perceptions as invalid usability 
data. 

Studies such as those by Bailey (1995), Tractinsky (1997), and Morris and Dillon (1997), however, began 
to address the discrepancy between performance and perceptions and to emphasize the importance of assessing 
perceptions. Since this kind of study is relatively new to the usability field, there is no standardized term to refer to 
the concept of perceptions. Words such as “preference” (Bailey 1995), “apparent usability” (Tractinsky, 1997), and 
“impression” (Morris & Dillon, 1997) are used interchangeably with “perceptions.” According to Tractinsky 
(1997), people formulate preferences for one system over the other on the basis of their vague beliefs about which 
interface would provide the fastest performance or apparent usability that they perceive from the aesthetics of the 
system. The preferred system, however, does not always result in better user performance or actual usability. In 
their study on Netscape, Morris and Dillon (1997) conclude that users' initial perceptions of Netscape’s usefulness 
and ease of use significantly influence their attitude toward using Netscape as well as their intention to use it. The 
implication of these studies is that we need to treat performance and perceptions separately. They also tell us that 
we should examine users’ perceptions of the system’s usability as well as their behavioral performance while using 
the system, to measure overall usability accurately. 



Evaluation Methodology 

Perception -Based Usability Evaluation 
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In this study, we used a perception-based usability evaluation method. An EPSS like STEPS was a 
relatively new idea to the education community, so we were interested in investigating whether target users of 
STEPS liked the system and perceived it to be usable. In addition, STEPS was still under construction at the time of 
the evaluation so collecting performance data at that stage did not seem to be meaningful. 

This evaluation elicited participants’ perceptions of three aspects of the STEPS web site: usability of the six 
main menus, existence of necessary EPSS components, and effectiveness of nine main menu icons. The STEPS 
web site consisted of nine main menus and six of them were primary components of the site. The six menus 
included Main Menu, Lesson Architect, Best Practice, Sample Unit, Web Links, and Tutorial Library. The matter of 
our primary concern was to evaluate whether the participants perceived the six menus to be effective, efficient, and 
satisfactory (i.e., usable.) In addition, we were interested in assessing whether the participants perceived that STEPS 
contained all the necessary features of an EPSS. As described in the literature review section, an EPSS typically 
contains four basic components: tools, database, expert system, and instructions. Lastly, we wanted to evaluate the 
effectiveness of icons associated with the nine main menus. On the main page, each main menu was represented 
through the combination of an icon and a textual link underneath. Our concern regarded whether or not the icons 
matched the textual links, representing the content of their corresponding pages well. 

Data Collection Methods 

Evaluation instruments included a usability questionnaire, a components questionnaire, a matching 
worksheet, observations, and structured interviews. The usability questionnaire (See Appendix A) aimed to measure 
the participants’ perceptions of the usability of the six main menus. It consisted of yes -no answer items such as, 

"Are the page contents useful for intended users?" "Are the navigation icons or texts consistent?" and "Do you like 
using this system?" We used one questionnaire for each of the six main menus. The components questionnaire 
aimed to measure participants' perceptions of whether the STEPS website contained all the necessary EPSS 
components or not (See Appendix B). It consisted of questions like, "Does this EPSS have tools (e.g., templates, 
forms, word processor, spreadsheets) for facilitating teacher performance?" The matching worksheet (See Appendix 
C) assessed whether participants could match the main menu graphics with textual links correctly. We provided 
participants with a list of icons and texts and asked them to match corresponding icon and text. While the 
participants were exploring the site and responding to the questionnaires and the matching worksheet, observations 
were made to collect participants’ reactions such as expressions of their frustration. The structured interviews (See 
Appendix D) at the end of the evaluation aimed to triangulate data from the questionnaires and observations. 

To ensure the trustworthiness of the evaluation, we employed triangulation and member-checking 
procedures. This evaluation used three kinds of triangulation techniques: (1 ) data triangulation through the use of 
multiple data sources (e.g., questionnaires, observations, and interviews), (2) participant triangulation by asking 
individuals with diverse backgrounds to evaluate the website (e.g., elementary school teachers, instructional 
designers, and professors), and (3) method triangulation through the use of various data collection methods (e.g., 
questionnaires, observations, matching worksheets, and interviews. After analyzing observation and interview data, 
we summarized the participants’ opinions and then asked the participants to review the summaries. The aim was to 
ensure that the data analysis results matched their original opinion. 

Participants 

Five participants engaged in the evaluation. Virzi (1992) conducted experiments regarding sample size for 
usability studies and concluded that observing four or five participants will allow a practitioner to discover 80% of a 
product's usability problems and observing additional participants will reveal increasingly fewer new usability 
problems. In addition, it is well known in the qualitative case study literature that there is a certain point after which 
discovery of new findings reaches saturation. 

The target users of the STEPS website were K-12 teachers. Therefore, we invited people who had teaching 
backgrounds to participate; three of them have taught in K-12 settings and two have worked as teacher educators. 

We also invited people who had expertise in design and development of computer-assisted instructional systems, in 
addition to teaching experience. The purpose was to gather expert opinion on the design of the website. Appendix 
E summarizes profiles of the participants. 

Evaluation Procedures 

The evaluation took place in the most typical computing environment of each participant. Some 
participants did computing at home and others worked at the computer labs provided by their university. All 
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participants used LAN Internet connection provided by the university. They used an IBM compatible PC or a 
notebook that had 24 RAM or more memory, Pentium 133 or faster processor, and 12- or 14-inch monitors. 

At the beginning of the evaluation, the participants filled out a Demographic Information questionnaire 
(See Appendix F.) Then they were given instructions about how to conduct the evaluation. They were instructed to 
cypress any criticism frankly, use a think-aloud technique, and feel free to ask any questions about the evaluation 
procedures. It was particularly emphasized that if they were having difficulties in using the website it was not their 
fault but the fault of the website’s ineffective design. When the participants felt comfortable about beginning the 
evaluation, we opened up the home page of the STEPS website and introduced the site's general purpose. Then, the 
participants were given six usability questionnaires, a components questionnaire, and a matching worksheet. They 
were requested to complete the questionnaires and the matching worksheet while exploring the site. Different 
participants looked at different levels of instruction since the entire STEPS website was too big for one person to 
explore in a given period of time. However, the evaluation criteria and scope were the same for all participants. 
Participant A explored the K3-5 level, participant B the K6-8, participant C the K9-12, participant D the K3-5, and 
participant E the K3-5. The participants thought out loud while exploring the site and we took notes of their 
comments. We sometimes asked probing questions if they did not verbalize problems voluntarily. When they 
finished all the questionnaires and the worksheet, a structured interview was conducted. 

Evaluation Results 

Results were derived from three evaluations: 1) evaluation of the six main menus’ usability using 
questionnaires, observations, and interviews; 2) evaluation of the EPSS corrponents using a questionnaire; and 3) 
evaluation of the match between icons and texts using a matching worksheet. 

Usability of six main menus 

The usability questionnaires asked for participants’ perception regarding each menu’s effectiveness, 
efficiency, and appeal. Table 1 summarizes percentages of positive (i.e., yes) responses. 



Table 1. Percentage of positive responses in six usability questionnaires 





Best 


Sample 


Web 


Tutorial 


Lesson 


Main 




Practice 


Unit 


Links 


Library 


Architect 


Menu 


Effectiveness 

Are the page contents useful for intended 


100 


100 


80 


60 


20 


0 


users? 

Do you believe this site can facilitate teacher 


100 


80 


60 


60 


40 


0 


performance? 

Mean 


100 


90 


70 


60 


30 


0 


Efficiency 

Are the navigation icons or texts consistent? 


80 


60 


0 


0 


20 


0 


Is it easy to navigate back and forth? 


80 


80 


0 


100 


0 


0 


Is the screen design user-friendly? (e.g.. 


80 


60 


20 


20 


20 


0 


letter size, color, graphics, etc.) 
Is the information concise? 


80 


40 


40 


20 


20 


20 


Are the sentences easy to understand? 


80 


100 


100 


100 


100 


20 


Mean 


80 


60 


35 


48 


32 


8 


Appeal 

Are you interested in this EPSS? 


100 


60 


60 


60 


20 


0 


Do you like using this system? 


100 


60 


60 


60 


40 


0 


Mean 


100 


60 


60 


60 


30 


0 



In the effectiveness evaluation, participants responded most positively to Best Practice. Sample Unit was 
second at 90%, followed by Web Links, Tutorial Library, Lesson Architect, and Main Menu. In the efficiency test. 
Best Practice was rated to be the most positive followed by Sample Unit, Tutorial Library, Web Links, and Lesson 
Architect, with Main Menu receiving the least positive evaluation. In the appeal evaluation. Best Practice received 
the most positive responses, followed by Sample Unit, Web Links, Tutorial, Lesson Architect, and Main Menu. 
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Overall, the participants responded to the three criteria in a similar pattern. They favored Best Practice the most and 
Main Menu the least for all three criteria. 

The results from the interviews were congruent with those of the questionnaires. Most participants made 
positive comments on Best Practice, including “menus organized by subject makes it easy to grasp the content,” “it 
provides examples of specific objectives,” “table structure of the menu makes it easy to navigate,” “color change 
gives the indications of where I am,” and “it provides concise information with graphics and animations.” For 
Sample Unit, participants made both positive and negative comments. On the other hand, participants responded 
more negatively to Main Menu and Lesson Architect. For Main Menu, in particular, five participants expressed 
similar concerns such as “the location of buttons is inconsistent so navigation is not easy,” and “after clicking, there 
is no consistency in screen display.” For Lesson Architect, participants responded negatively, making comments 
such as, “it’s hard to know the function of icon-only buttons,” “there is no direction after clicking,” “too much 
scrolling to do,” and “data are not organized so it’s not easy to find needed information.” 

EPSS components 

Most participants identified STEPS as a system that contained database and tool components more than 
instruction or expert system components. All participants agreed that STEPS contained database, 80% agreed it 
contained tools, 40% instruction, and 20% expert system. They pointed out that Lesson Architect was a template 
embedded with word processing functions. They commented that, using Lesson Architect, a teacher could develop 
and save instructional designs and curriculum plans with the help of instructional theories such as Gauge’s Event of 
Instruction. Web Links included links to numerous web sites. Best Practice was a database of effective classroom 
activities. 

Although participants did not recognize it clearly, STEPS did include expert system and instruction 
components. Ask Coach and Tutorial Library were representations of the components. Ask Coach provided field- 
sensitive help in a pop-up window; however, most participants stated that the help was not informative or useful for 
actual instructional activities. The fact that Ask Coach was still under construction might have been the cause of this 
response. Also, most users thought that it is very limited in that it only provides fixed explanations and cannot make 
a relational query within the Coach or support customized help depending on users’ level of competence. As for the 
Tutorial Library, participants responded that the information was not useful for specific design activities. 

Match between icon and text 

None of the participants matched more than four of the nine pairs. Evaluator A matched 3 pairs, evaluator 
B matched 2, Evaluator C, 1, Evaluator D, 4, and Evaluator E, 3. The difficulties in matching icons with texts were 
caused mainly by the inadequacy of icons in portraying corresponding textual information. In addition, the way line 
spacing was used made it difficult for participants to know which icon corresponded with which text. 

Overall comments 

The participants agreed unanimously that an EPSS like STEPS was a powerful tool for effective 
instruction. However, they said that STEPS should go beyond being an electronic book. They commented that it 
should facilitate interaction between teachers and provide guidelines, not just information. They thought teachers 
would be able to enhance their performance with the help of such interaction and guidelines. 

Recommendations for Designing an E-EPSS Effectively 

Based upon our evaluation results and literature reviews on E-EPSS design, we recommend the following 
heuristics for effective E-EPSS design. 

Design an interface that supports teacher performance 

Participants did not think that Main Menu and Lesson Architect facilitated teacher performance. This 
response is critical in the case of Lesson Architect especially, because Lesson Architect is supposed to facilitate 
teacher performance by helping teachers to build lesson plans. Laffey (1995) asserts that current models of EPSSs 
provide the kinds of resources needed but do not support the processes by which these resources are used or 
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customized for the work environment. Instead of seeing an EPSS as merely a vehicle for delivering information, we 
need to see it as a re -conceptualization of the work environment. 

Provide context specific information 

Participants thought STEPS lacked expert system functions and particularly context - or subject-specific 
guidelines. Evaluator B, for example, responded, “STEPS looks like a teacher guidebook but its information is too 
general, not subject-specific.” She recommended that the Language Arts section should contain more specific 
information to be useful in actual instructional design activities. 

The process of determining the content of an E-EPSS should involve a thorough examination of related 
literature and consultation with subject matter experts, teachers, and teacher educators. It should also include 
identification of teachers’ daily activities such as lesson planning and instruction design. Once the activities are 
identified, the designer needs to specify possible questions teachers could ask at each stage of the activity, and 
provide necessary supports that will answer the questions. This kind of support must be updated and expanded when 
necessary. 

Provide structured navigational schemes 

The evaluation results show that people like structured navigational scheme such as menus organized by 
subject in a table format. To provide a structured navigational scheme, it is important to build a “bird’s-eye view” 
into the design. Users should be informed about where they are now and where they should go next . In STEPS, the 
hierarchy and relationships among different levels (corporation level, unit level, and lesson level) are not indicated 
clearly, resulting in confusion in navigation. 

The Object Action Interface (OAI) model by Shneiderman (1998) is useful for designing a structured 
interface. It guides system designers to view interfaces in terms of the tasks that the interfaces will carry out. By 
matching interfaces with tasks, designers can produce task-oriented and structured interface designs. 

Consider the level of students the users teach 

Participants mentioned that content in STEPS should consider grade levels. If the content is for elementary 
teachers, for example, it should reflect perspectives and difficulty levels appropriate for elementary students. In this 
way, the teachers will be able more easily to relate what they get through STEPS to their students’ learning 
activities. Rummers (1998) contends that the structure of an EPSS should be easy to use, flexible, and tailored for 
end-users with different needs and expertise. 

Provide interactive screen designs 

Participants made comments such as, “it’s more like a technical report,” “texts are not easy to read,” “the 
content seems like texts from a book,” and “I need to do too much scrolling and get lost often.” These comments 
tell us that STEPS could be perceived as an electronic page-turner. To prevent this perception, STEPS needs to 
incorporate action-oriented design approaches such as anchoring tools in the task domain and supporting error 
recognition and recovery, and user performance. 

Limitation of study 

Although we referred to a published article by the STEPS developers, Northrup and Pilcher (1998), we did 
not have access to enough of the developers’ opinions regarding their intentions about STEPS development and the 
purpose of the program. In addition, we did not have any information about their development processes such as 
difficulties in the processes and timelines of their development and implementation. 

The evaluation materials that we developed for this study are the results of our research on usability and 
educational evaluation studies. Although we hope the materials helped us measure participants’ perceptions 
accurately, we think the materials need to be subjected to quality examination such as reliability and validity tests. 

It should be noted that the questionnaires measured participants’ subjective perceptions of the program. 
Objective performance measures such as time taken to finish certain tasks or error rate were not collected in this 
study. A future evaluation study that collects performance data will provide us with a richer picture of the 
program’s usability and educational value. 
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